Method for simulating flow of an extinguishing agent

ABSTRACT

Method and apparatus are presented for an augmented reality-based firefighter training system. The system includes hardware for motion tracking, display, and vari-nozzle instrumentation. System software includes a real-time fire model, a layered smoke obscuration model, simulation of an extinguishing agent, and an interface to a zone fire model. Physical modeling and graphical elements in the software combine to create realistic-looking fire, smoke, and extinguishing graphics. The hardware and software components together contribute to a realistic, interactive training experience for firefighters.

[0001] This application is a divisional of pending Nonprovisional patentapplication Ser. No. 09/525,983 filed on Mar. 15, 2000.

CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority of pending Provisional patentapplications 60/124,428, filed on Mar. 15, 1999, 60/142,120, filed onJul. 2, 1999, 60/145,401, filed on Jul. 23, 1999, and 60/147,725, filedon Aug. 6, 1999.

GOVERNMENT RIGHTS CLAUSE

[0003] This invention was made with Government support under ContractNumber N-61339-98-C-0036 awarded by the Naval Air Warfare CenterTraining Systems Division of Orlando, Fla. The Government has certainrights in the invention.

FIELD OF THE INVENTION

[0004] This invention relates to training firefighters in an augmentedreality (AR) simulation that includes creation of graphics depictingfire, smoke, and application of an extinguishing agent; and displayingthe simulated phenomena anchored to real-world locations seen through ahead-worn display.

COPYRIGHT INFORMATION

[0005] A portion of the disclosure of this patent document containsmaterial that is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument or the patent disclosure as it appears in the Patent andTrademark Office records but otherwise reserves all copyright workswhatsoever.

BACKGROUND OF THE INVENTION

[0006] Current fire simulation for firefighter training is accomplishedat facilities that use propane burners and extinguishing agentcollectors to simulate the behavior of various types of fires. Thisapproach presents numerous disadvantages, such as safety risksattributable to unintended reflash and explosion; environmental damageattributable to combustion byproducts; health risks to crews due toinhalable combustion byproducts; high operation costs attributable tofuel requirements; high maintenance costs to ensure system integrity andsafety; and unrealistic fire simulations for some types of fires (allsimulations appear as propane fires as opposed to oil, electrical orpaper; and simulated smoke is white instead of black).

[0007] A need exists for a new generation of fire fighting/damagecontrol simulation system which does not use live fires. These systemsmust be capable of providing a high fidelity representation of the smokeand flames, as well as a realistic representation of the environment (toinclude fellow crew members). Augmented reality (AR) technology allowsoverlay of computer-generated graphics on a person's view of the realworld. With AR, computer generated fire, smoke, and extinguishing agentscan safely replace live fire training while still allowing trainees toview and interact with each other and the real-world environment. Thisallows safe, cost-effective training with greater realism than purevirtual reality (VR) simulations.

[0008] The majority of current generation of fire fighting trainingsystems use live, propane-based fires which are unsafe, particularly foruse in contained areas such as onboard ships, and in real structures. Ina training environment, the use of live propane-based fires presentssafety, health and environmental risks.

[0009] The primary objective of this invention is the development of anaugmented reality-based training (ARBT) system for fire fighting, withapplication to rescue and hazardous material mitigation. In fact, in anyfire situation there are multiple goals, including:

[0010] Search, rescue, and extrication

[0011] Ingress into, and egress from, a structure

[0012] Fire suppression

[0013] Structure stabilization

[0014] Team coordination—command & control

[0015] Fire cause determination

[0016] In each of the goals, firefighters engage in a number ofcognitive and physical tasks critical to the survival of both firevictims and firefighters, as well as to the timely suppression of afire. Tasks that fall under this category are

[0017] (1) Navigation

[0018] (2) Situation awareness

[0019] (3) Decision making/problem solving

[0020] (4) Stress management

[0021] These tasks are undertaken, usually in concert with one another,to achieve the above goals. Training in these four tasks provides thefoundation for a firefighter to combat any fire situation. Anopportunity exists to develop an ARBT system which educates firefightersin these tasks in a safe and potentially less expensive environment, inalmost any location.

[0022] It is important at this juncture to distinguish between theconcept of reaction versus interaction with fire and smoke. By reactionwe connote responses made by a firefighter to conditions caused by fireand smoke; in this situation he/she does not alter the evolution of thefire and smoke. By interaction we mean that the firefighter directlyaffects the evolution of the fire and smoke by such actions as firesuppression and ventilation. As stated above, Tasks (1) to (4) areapplicable to any fire situation—reactive or interactive. Therefore, anysignificant improvement in developing training skills for Tasks (1) to(4) will result in a significantly skilled firefighter for both reactiveand interactive scenarios.

SUMMARY OF THE INVENTION

[0023] An objective of this invention is to demonstrate the feasibilityof augmented reality as the basis for an untethered, ARBT system totrain firefighters. Two enabling technologies will be exploited: aflexible, wearable belt PC and an augmented reality head-mounted display(HMD).

[0024] Unlike traditional augmented reality systems in which anindividual is tied to a large workstation by cables from head mounteddisplays and position trackers, the computer technology is worn by anindividual, resulting in an untethered, augmented reality system.

[0025] Augmented reality is a hybrid of a virtual world and the physicalworld in which virtual stimuli (e.g. visual, acoustic, thermal,olfactory) are dynamically superimposed on sensory stimuli from thephysical world.

[0026] This invention demonstrates a foundation for developing aprototype untethered ARBT system which will support the critical firefighting tasks of (1) navigation, (2) situation awareness, (3) stressmanagement, and (4) problem solving. The system and method of thisinvention can be not only a low-cost training tool for fire academiesand community fire departments, but also provides a test bed forevaluating future fire fighting technologies, such as decision aids,heads-up displays, and global positioning systems for the 21st centuryfirefighter.

[0027] Accordingly, the primary opportunity for an ARBT system is thetraining of firefighters in the areas of Tasks (1) to (4) above forreactive scenarios.

Significance of the Opportunity

[0028] Overall Payoffs. The inventive ARBT system has the significantpotential to produce

[0029] Increased safety

[0030] Increased task performance

[0031] Decreased workload

[0032] Reduced operating costs

[0033] A training program that aims to increase skills in the Tasks (1)to (4) is adaptable to essentially any fire department, large or small,whether on land, air, or sea.

[0034] Opportunities for Augmented Reality for Training. Augmentedreality has emerged as a training tool. Augmented reality can be amedium for successful delivery of training. The cost of an effectivetraining program built around augmented reality-based systems arisesprimarily from considerations of the computational complexity and thenumber of senses required by the training exercises. Because of thevalue of training firefighters in Tasks (1) to (4) for any firesituation, and because the program emphasizes firefighter reactions to(vs. interactions with) fire and smoke, training scenarios can beprecomputed.

[0035] As described elsewhere in this document, models exist which canpredict the evolution of fire and smoke suitable for trainingapplications. An opportunity exists to exercise these models off line tocompute reactive fire fighting scenarios. These precomputations can layout various fire-and-smoke induced phenomena which evolve dynamically intime and space and can produce multi-sensor stimuli to the firefighterin 3D space. (For example, if the firefighter stands up, he/she may findhis/her visibility reduced due to smoke, whereas if he/she crawls,he/she can see more clearly.)

[0036] It has been demonstrated that PC technology is capable ofgenerating virtual world stimuli—in real time. We can then apply ouraugmented reality capabilities to the development of an augmentedreality-based training system.

[0037] In summary, the opportunity identified above—which has focused onreactions of firefighters to fire and smoke in training scenarios—isamenable to augmented reality.

[0038] Opportunities for Augmented Reality for Training. In augmentedreality, sensory stimuli from portions of a virtual world aresuperimposed on sensory stimuli from the real world. If we consider acontinuous scale going from the physical world to completely virtualworlds, then hybrid situations are termed augmented reality.

[0039] The position on a reality scale is determined by the ratio ofvirtual world sensory information to real world information. Thisinvention creates a firefighter training solution that builds on theconcept of an augmented physical world, known as augmented reality.Ideally, all training should take place in the real world. However, dueto such factors as cost, safety, and environment, we have moved some orall of the hazards of the real world to the virtual world whilemaintaining the critical training parameters of the real world, e.g., weare superimposing virtual fire and smoke onto the real world.

[0040] For a fire example, consider the following. Suppose an officeroom fire were to be addressed using augmented reality. In this problem,a real room with real furniture is visible in real time through a headmounted display (HMD) with position tracker. Virtual fire and smoke dueto virtual combustion of office furniture can be superimposed on the HMDview of the physical office without ever having to actually ignite apiece of real furniture.

[0041] The inventive approach allows the firefighter to both react andinteract with the real world components and the virtual components ofthe augmented reality. Examples of potential real-world experiences tobe offered by our approach are given below in Table 1-1.

[0042] Clearly, simulation of training problems for firefighters cancomprise both physical and virtual elements. In many instances augmentedreality may be a superior approach when compared to completely virtualreality. For example, exercise simulators such as stationary bicycles,treadmills or stair climbing machines do not adequately capture eitherthe physical perception or the distribution of workload on themusculoskeletal systems that would be produced by actually walking orcrawling in the physical world. Additionally, a firefighter can seehis/her fellow firefighters, not just a computer representation as inpure virtual reality.

[0043] Opportunities for Self-Contained Augmented Reality. A low-cost,flexible, wearable belt PC technology may be used in augmented realityfirefighter training. This technology, combined with augmented realityand precomputed fire scenarios to handle tasks (1) to (4) above forvarious physical locations, allows a firefighter to move untetheredanywhere, anytime, inexpensively and safely. This will significantly addmore realistic training experiences.

[0044] Background Review of Fire Simulation. Mitler (1991) divides firemodels into two basic categories: deterministic and stochastic models.Deterministic models are further divided into zone models, field models,hybrid zone/field models, and network models. For purposes ofpracticality and space limitations, we limit the following discussionsto deterministic models, specifically zone type fire models. Mitler goeson to prescribe that any good fire model must describe convective heatand mass transfer, radiative heat transfer, ignition, pyrolysis and theformation of soot. For our purposes, models of flame structure are alsoof importance.

[0045] Zone models are based on finite element analysis (FEA). In a zonemodel of a fire, a region is divided into a few control volumes—zones.The conditions within each volume are usually assumed to beapproximately constant. In the study of compartment fires, two or morezones typically are used: an upper layer, a lower layer, and,optionally, the fire plume, the ceiling, and, if present, a vent. Zonemodels take the form of an initial value problem for a system ofdifferential and algebraic equations. Limitations of zone models includeambiguity in the number and location of zones, doubt on the validity ofempirical expressions used to describe processes within and betweenzones, and inapplicability of zones to structures with large area orcomplex internal configurations.

[0046] For many training applications, such effects are not significantfor purposes of this invention. Friedman (1992) performed a survey offire and smoke models. Of 31 models of compartment fire, Friedman found21 zone models and 10 field models. Most of the zone models can run on aPC, while most of the field models require more powerful computationalresources.

[0047] Background Review of Virtual Reality-Based Training—and Potentialfor Augmented Reality-Based Training. Probably the core issuesurrounding the development of any training system or program is theefficiency of the transfer of knowledge and skills back into theworkplace. Individual development ultimately rests on the ability toadapt acquired skills to novel situations. This is referred to, by some,as a metaskill. The transference of skills and the building ofmetaskills are fundamental concepts against which virtual reality mustbe considered for its suitability as a basis for the delivery oftraining.

[0048] Experiential learning is based on the premise that people bestlearn new skills by successfully performing tasks requiring thoseskills. The application of virtual reality to the delivery of trainingbuilds on the promise of experiential learning to maximize the transferof training into the task environment. Furthermore, virtual realityinterfaces also hold the potential for being more motivating thantraditional training delivery media by making the training experienceitself more fun and interesting. Augmented reality retains thesestrengths while providing a real world experience for the firefighter.

[0049] When concerned with the transfer of skills from a virtual worldto the real world, the issue of virtual world fidelity is often raised.Alessi (1988) examined the issue of simulator fidelity in both initiallearning and transfer of learning and found that the impact of simulatorfidelity increases with the level of expertise of the student. He goeson to recommend that fidelity increase along lines of instructionphases: presentation, guidance, practice, and assessment. Alessi'sresults are corroborated by Lintern et al. (1990) in their work on thetransfer of training using flight simulators for initial training oflanding skills. Most notably the authors found that feedback, related tocorrect performance of the landing task, resulted in increased transferof training. They also found that transfer of training did notnecessarily increase with increasing simulator fidelity. These resultson fidelity are important in that they emphasize that simply creating atask environment in a virtual world without consideration of learningprocesses may not be sufficient to transfer skills to the physicalworld.

[0050] Review of Augmented Reality Equipment. A description of augmentedreality was presented above. Commercial off the shelf technologies existwith which to implement augmented reality applications. This includeshelmet-mounted displays (HMDs), position tracking equipment, andlive/virtual mixing of imagery.

BRIEF DESCRIPTION OF THE DRAWINGS

[0051]FIG. 1 is a block diagram indicating the hardware components of anembodiment of the augmented reality (AR) firefighter training system,also useful in the method of the invention.

[0052]FIG. 2A illustrates the geometric particle representationassociated with smoke.

[0053]FIG. 2B illustrates the geometric particle representationassociated with flames.

[0054]FIG. 2C illustrates the three particle systems used to represent afire.

[0055]FIG. 3 illustrates the idea of two-layer smoke obscuration.

[0056]FIG. 4A illustrates particle arrangement for a surfacerepresentation of a particle system.

[0057]FIG. 4B illustrates a surface mesh for a surface representation ofa particle system.

[0058]FIG. 5 illustrates the technologies that combine to create an ARfirefighter training system, and method.

DETAILED DESCRIPTION OF THE INVENTION

[0059]FIG. 1 is a block diagram indicating the hardware components ofthe augmented reality (AR) firefighter training system, also useful forthe method of the invention. Imagery from a head-worn video camera 4 ismixed in video mixer 3 via a linear luminance key withcomputer-generated (CG) output that has been converted to NTSC usingVGA-to-NTSC encoder 2. The luminance key removes white portions of thecomputer-generated imagery and replaces them with the camera imagery.Black computer graphics remain in the final image, and luminance valuesfor the computer graphics in between white and black are blendedappropriately with the camera imagery. The final image is displayed to auser in head-mounted display (HMD) 5.

[0060] One alternative to the display setup diagrammed in FIG. 1 is theuse of optical see-through AR. In such an embodiment, camera 4 and videomixer 3 are absent, and HMD 5 is one that allows its wearer to seecomputer graphics overlaid on his/her direct view of the real world.This embodiment is not currently preferred for fire fighting becausecurrent see-through technology does not allow black smoke to obscure aviewer's vision.

[0061] A second alternative to the display setup diagrammed in FIG. 1 iscapturing and overlaying the camera video signal in the computer, whichremoves the video mixer 3 from the system diagram. This allowshigh-quality imagery to be produced because the alpha, or transparencychannel of the computer 1 graphics system may be used to specify theamount of blending between camera and CG imagery. This embodiment is notcurrently preferred because the type of image blending described hererequires additional delay of the video signal over the embodiment ofFIG. 1, which is undesirable in a fire fighting application because itreduces the level of responsiveness and interactivity of the system.

[0062] A third alternative to the display setup diagrammed in FIG. 1 isproducing two CG images and using one as an external key for luminancekeying in a video mixer. In this embodiment, two VGA-to-NTSC encoders 2are used to create two separate video signals from two separate windowscreated on the computer 1. One window is an RGB image of the scene, anda second window is a grayscale image representing the alpha channel. TheRGB image may be keyed with the camera image using the grayscale alphasignal as the keying image. Such an embodiment allows controllabletransparency with a minimum of real-world video delay.

[0063]FIG. 1 diagrams the two 6 degree-of-freedom (6DOF) trackingstations 7 and 8 present in all embodiments of the system. One trackingstation 7 is attached to the HMD 5 and is used to measure a user's eyelocation and orientation in order to align the CG scene with the realworld. In addition to matching the real-world and CG eye locations, thefields of view must be matched for proper registration. The secondtracking station 8 measures the location and orientation of a nozzle 9that may be used to apply virtual extinguishing agents. Prediction ofthe 6DOF locations of 7 and 8 is done to account for system delays andallow correct alignment of real and virtual imagery. The amount ofprediction is varied to allow for a varying CG frame rate. The systemuses an InterSense IS-600 tracking system 6, and it also supports theInterSense IS-900 and Ascension Flock of Birds.

[0064] Additional hardware is attached to vari-nozzle 9 to allow controlof a virtual water stream. A potentiometer attached to the bail handlemeasures the handle angle to determine whether or not the nozzle is onor off. A second potentiometer attached to the pattern selector measuresthe nozzle spray angle. An analog-to-digital (A/D) converter readsvoltages from the potentiometers and converts them into appropriateunits for nozzle control.

[0065] A method for real-time depiction of fire is diagrammed in FIG.2A-C. A particle system is employed for each of the persistent flame,intermittent flame, and buoyant plume components of a fire, asdiagrammed in FIG. 2C. The particles representing persistent andintermittent flames are created graphically as depicted in FIG. 2B. Fourtriangles make up a fire particle, with transparent vertices 12-15 atthe edges and an opaque vertex 16 in the center. Smooth shading of thetriangles interpolates vertex colors over the triangle surfaces. Thelocal Y axis 27 of a fire particle is aligned to the direction ofparticle velocity, and the particle is rotated about the local Y axis 27to face the viewer, a technique known as “billboarding.” A fire texturemap is projected through both the persistent and intermittent flameparticle systems and rotated about a vertical axis to give a horizontalswirling effect.

[0066] Smoke particles, used to represent the buoyant plume portion of aflame, are created graphically as depicted in FIG. 2A. A texture map 11representing a puff of smoke is applied to each particle 10, whichconsists of two triangles, and transparency of the texture-mappedparticle masks the appearance of polygon edges. Smoke particles 10 arerotated about two axes to face the viewer, a technique known as“spherical billboarding.”

[0067] The flame base 17 is used as the particle emitter for the threeparticle systems, and buoyancy and drag forces are applied to eachsystem to achieve acceleration in the persistent flame, near-constantvelocity in the intermittent flame, and deceleration in the buoyantplume. An external force representing wind or vent flow may also beapplied to affect the behavior of the fire plume particles. When flameparticles are born, they are given a velocity directed towards thecenter of the fire and a life span inversely proportional to theirinitial distance from the flame center. The emission rate ofintermittent flame particles fluctuates sinusoidally at a ratedetermined by a correlation with the flame base area. Flame height maybe controlled by appropriately specifying the life span of particles inthe center portion of the flame.

[0068] A number of graphical features contribute to the realisticappearance of the fire and smoke plume diagrammed in FIG. 2C. Depthbuffer writing is disabled when drawing the particles to allow blendingwithout the need to order the drawing of the particles from back tofront. A light source is placed in the center of the flames, and itsbrightness fluctuates in unison with the emission rate of theintermittent flame particle system. The light color is based on theaverage color of the pixels in the fire texture map applied to the flameparticles. Lighting is disabled when drawing the flame particles toallow them to be at full brightness, and lighting is enabled whendrawing the smoke particles to allow the light source at the center ofthe flame to cast light on the smoke plume. A billboarded,texture-mapped, polygon with a texture that is a round shape fading frombright white in the center to transparent at the edges is placed in thecenter of the flame to simulate a glow. The RGB color of the polygon isthe same as the light source, and the alpha of the polygon isproportional to the density of smoke in the atmosphere. When smoke isdense, the glow polygon masks the individual particles, making theflames appear as a flickering glow through smoke. The glow width andheight is scaled accordingly with the flame dimensions.

[0069]FIG. 3 describes the concept of two layers of smoke in acompartment. In a compartment fire, smoke from the buoyant plume risesto the top of a room and spreads out into a layer, creating an upperlayer 20 and a lower layer 21 with unique optical densities. The lowerlayer has optical density k₁, and the upper layer has density k₂.Transmittance through the layers from a point P 19 on the wall to aviewer's eye 18 is given by the equation T=e^(−(k) ^(₁) ^(x) ^(₁) ^(+k)^(₂) ^(x) ^(₂) ). The color of 19 as seen through smoke is given byC=TC_(i)+(1−T)C_(s) , where C_(i) represents the color of 19 with noobscuration and C_(s) represents the smoke color.

[0070] To apply the concept of two-layer smoke in an AR system, apolygonal model of the real room and contents is created. The model isaligned to the corresponding real-world using the system of FIG. 1. Asthe model is drawn, the above equations are applied to modify the vertexcolors to reflect smoke obscuration. Smooth shading interpolates betweenvertex colors so that per-pixel smoke calculations are not required. Ifthe initial color, C_(i), of the vertices is white, and the smoke color,C_(s), is black, the correct amount of obscuration of the real worldwill be achieved using the luminance keying method described above. Inthe other video-based embodiments, the above equations can be applied tothe alpha value of vertices of the room model.

[0071] In computer graphics, color values are generally specified usingand integer range of 0 to 255 or a floating point range of 0 to 1.0.Using the obscuration approach described above of white objects thatbecome obscured by black smoke, this color specification does not takeinto account light sources such as windows to the outdoors, overheadfluorescent lights, or flames; which will shine through smoke more thannon-luminous objects such as walls and furniture. To account for this, aluminance component was added to the color specification to affect howobjects are seen through smoke. Luminance values, L, range from 0 to 1.0in this embodiment, and they alter the effective optical density asfollows: k′=k(1−L). This makes objects with higher luminance showthrough smoke more than non-luminous (L=0) objects.

[0072] One additional component to the layered smoke model is theaddition of a smoke particle system, as depicted in FIG. 2A. A smokeparticle system is placed in the upper, denser layer 20 to give movementto the otherwise static obscuration model. To determine the volume andoptical density of the upper smoke layer, one method is to assign volumeand density characteristics to the buoyant plume smoke particles. When abuoyant plume smoke particle fades after hitting the ceiling of a room,the volume and optical density of the particle can be added to the upperlayer to change the height and optical density the layer.

[0073] The same polygonal model used for smoke obscuration is also usedto allow real-world elements to occlude the view of virtual objects suchas smoke and fire. A fire plume behind a real desk that has been modeledis occluded by the polygonal model. In the combined AR view, it appearsas if the real desk is occluding the view of the fire plume.

[0074] Graphical elements such as flame height, smoke layer height,upper layer optical density, and lower layer optical density may begiven a basis in physics by allowing them to be controlled by a zonefire model. A file reader developed for the system allows CFAST modelsto control the simulation. CFAST, or consolidated fire and smoketransport, is a zone model developed by the National Institute ofStandards and Technology (NIST) and used worldwide for compartment firemodeling. Upper layer temperature calculated by CFAST is monitored bythe simulation to predict the occurrence of flashover, or full roominvolvement in a fire. The word “flashover” is displayed to a traineeand the screen is turned red to indicate that this dangerous event inthe development of a fire has occurred.

[0075] A key component in a fire fighting simulation is simulatedbehavior and appearance of an extinguishing agent. In this embodiment,water application from a vari-nozzle 9, 23, and 25 has been simulatedusing a particle system. To convincingly represent a water stream withminimal computation, a surface representation of a particle system wasdevised. This representation allows very few particles to represent awater stream, as opposed to alternative methods that would require theentire volume of water to be filled with particles. Behavior such asinitial water particle velocity and hose stream range for differentnozzle settings is assigned to a water particle system. Water particlesare then constrained to emit in a ring pattern from the nozzle locationeach time the system is updated. This creates a series of rings ofparticles 22 as seen FIG. 4A. The regular emission pattern and spacingof particles allows a polygon surface to easily be created using theparticles as triangle vertices, as seen in the wireframe mesh 24 in FIG.4B. The surface 24 is texture-mapped with a water texture, and thetexture map is translated in the direction of flow at the speed of theflow. A second surface particle system that is wider than the first isgiven a more transparent texture map to the hard edge of the surfaceparticle system representation. A third particle system using smallbillboards to represent water droplets is employed to simulate watersplashing.

[0076] To add realism to the behavior of the water stream, collisiondetection with the polygonal room and contents model is employed. A rayis created from a particle's current position and its previous position,and the ray is tested for intersection with room polygons to detectcollisions. When a collision between a water particle and room polygonis detected, the particle's velocity component normal to the surface isreversed and scaled according to an elasticity coefficient. The samecollision method is applied to smoke particles when they collide withthe ceiling of a room. Detection of collision may be accomplished in anumber of ways. The “brute force” approach involves testing everyparticle against every polygon. For faster collision detection, a spacepartitioning scheme may be applied to the room polygons in apreprocessing stage to divide the room into smaller units. Particleswithin a given space are only tested for collision with polygons thatare determined to be in that space in the preprocessing stage. Somespace partitioning schemes include creation of a uniform 3-D grid,binary space partitioning (BSP), and octree space partitioning (OSP).

[0077] A simpler approach to collisions that is applicable in an emptyrectangular room is the use of an axis-aligned bounding box. In such animplementation, particles are simply given minimum and maximum X, Y, andZ coordinates, and a collision is registered if the particle positionmeets or exceeds the specified boundaries.

[0078] To increase the realism of water application, steam is generatedwhen water particles collide at or near the location of the fire. Steamparticle emitters are placed at the collision locations and they aregiven an emittance rate that is scaled by the size of the fire and theinverse of the collision's distance from the fire. Steam particles arerendered as spherically billboarded, texture-mapped polygons similar tothe smoke particles in FIG. 2A, but with a different texture map 11 anddifferent particle behavior. In compartment fire fighting, steam isgenerated when a hose stream is aimed at the upper, hot gas layer. Steamparticle systems may be placed in this layer to simulate thisphenomenon. Steam emittance in the upper layer can be directlyproportional to the temperature of the upper layer as calculated byCFAST.

[0079] To simulate extinguishment, a number of techniques are employed.Water particles that collide with the surface on which the flame base islocated are stored as particles that can potentially contribute toextinguishment. The average age of these particles is used inconjunction with the nozzle angle to determine the average water densityfor the extinguishing particles. Triangles are created using theparticle locations as vertices. If a triangle is determined to be on topof the fire, then an extinguishment algorithm is applied to the fire.

[0080] Extinguishing a fire primarily involves reducing and increasingthe flame height in a realistic manner. This is accomplished by managingthree counters that are given initial values representing extinguishtime, soak time, and reflash time. If intersection between water streamand flame base is detected, the extinguish time counter is decremented,and the flame height is proportionately decreased until both reach zero.If water is removed before the counter reaches zero, the counter isincremented until it reaches its initial value, which increments theflame height back to its original value. After flame height reacheszero, continued application of water decrements the soak counter untilit reaches zero. If water is removed before the soak counter reacheszero, the reflash counter decrements to zero and the flames re-igniteand grow to their original height. The rate at which the extinguish andsoak counters are decremented can be scaled by the average water densityfor more realistic behavior.

[0081] To allow more realistic extinguishing behavior, a flame base isdivided into a 2-D grid of smaller areas. Each grid square is an emitterfor three particle systems: persistent flames, intermittent flames, andbuoyant plume. When flame particles are born in a grid square, they aregiven a velocity directed towards the center of the flame base and alife span inversely proportional to their initial distance from theflame center. This allows multiple flame particle systems to appear as asingle fire. Each grid square has an independent flame height,extinguish counter, soak counter, and reflash counter. This allowsportions of a flame to be extinguished while other portions continue toburn. This is especially useful for larger fires where the hose streamcan only be directed at one part of the fire at a time.

[0082] 3-D audio allows sound volume to diminish with distance from asound emitter, and it allow works with stereo headphones to givedirectionality to sounds. 3-D audio emitters are attached to the fireand the hose nozzle. The fire sound volume is proportional to physicalvolume of the fire.

[0083] Appendix A contains settings for the parameters of particlesystems used in the invention. These parameters are meant to beguidelines that give realistic behavior for the particles. Many of theparameters are changed within the program, but the preferred startingparameters for flames, smoke, steam, and water are listed in theappendix.

[0084] Approach to Untethered ARBT for Firefighters. The basicphilosophy behind the objectives herein for developing an untetheredARBT system for firefighters follows from a systems-based approach totraining system development. The essential steps in such an approachare:

[0085] Determine training goals and functions

[0086] Implement a development strategy

[0087] Perform training needs analysis

[0088] Assess training needs

[0089] Collect and analyze task data

[0090] Undertake training system development

[0091] Write training objectives

[0092] Construct criterion measures

[0093] Construct evaluative measures

[0094] Choose a delivery system

[0095] Select and sequence content

[0096] Select an instructional strategy

[0097] Develop augmented reality firefighter training system softwareand hardware

[0098] Develop/implement an accurate position tracking system

[0099] Develop/implement capability for mixing real and virtual imagery

[0100] Develop/implement capability for anchoring virtual objects in thereal world

[0101] Develop/implement models for occluding real objects by virtualobjects and virtual objects by real objects

[0102] Develop/implement technology to display augmented reality scenesto the firefighter

[0103] Develop/implement models for fire, smoke, water, and steam

[0104] Perform system integration of the above (See FIG. 5)

[0105] Establish training system validity

[0106] Test & evaluate

[0107] The opportunity identified above amounts to an assessment oftraining needs of firefighters tempered by the realities ofstate-of-the-art technologies.

[0108] The issues in this needs assessment include:

[0109] Sensory representations of fire and smoke

[0110] Real-time presentation of those sensory representations

[0111] Modeling of fire spread

[0112] Instructor authoring of fire training exercises

[0113] We consider pre-flashover compartment fires in an effort todemonstrate feasibility of our approach to a training system. Flashoverrefers to the point in the evolution of a compartment fire in which thefire transitions from local burning to involvement of the entirecompartment.

[0114] One of the key elements of our approach is the precomputation offire dynamics. We have elected to use a zone-type fire model. Azone-type fire model should provide sufficient accuracy for meeting ourtraining objectives. There are a number of zone models available,including the Consolidated Fire and Smoke Transport Model (CFAST) fromNIST and the WPI fire model from Worcester Polytechnic Institute, amongothers.

[0115] The outputs of a zone-type fire model can be extended to achievea visual representation of a compartment fire. TABLE 1-1 Examples ofReal World Actions and Augmented World Effects in a Fire FightingTraining Scenario. The Self-Contained Augmented Reality training systemof this invention will provide real-world experiences like these.REALITY AUGMENTED REALITY Feel real walls See virtual fire near the realceiling Turn real doorknobs Hear virtual roaring fire on other side ofdoor Experience virtual smoke pour out Crawl Keep below virtual smokelayer of 2 feet above floor Climb real stairs See the loss of visibilityat top of virtual smoke-filled stairway Maybe smell virtual smoke ifyour SCBA is ill-fitted View a real aircraft on See virtual smoke andfire and locate real rescue real tarmac of runway points Enter realaircraft, Navigate among virtual smoke and fire filled physicallyperform a fuselage right-hand search Attempt search and rescue whilevirtual fire pattern advances on your position Make error in right-handsearch and become disoriented Experience claustrophobia as time on yourSCBA runs out and visibility decreases Experience stress when your SCBA“malfunctions” (via instructor radio remote control)

Task 1. Infrastructure for Real-Time Display of Fire

[0116] Task Summary. The organization and structuring of information tobe displayed is as important as actual display processing for real-timedynamical presentation of augmented environments. As a firefighter movesthrough a scenario (using an augmented reality device) the location,extent, and density of fire and smoke change. From a computationalperspective, an approach is to examine the transfer of data to and fromthe hard disk, through system memory, to update display memory with thedesired frequency.

[0117] Approach to This Task. Precomputation of the bulk of afirefighter training simulation, implies that most of the operationsinvolved in real-time presentation of sensory information revolve arounddata transfer. In order to identify bottlenecks and optimize informationthroughput, it is advantageous to analyze resource allocation in thecontext of some systems model, such as queuing theory. Given such ananalysis, we may then implement data structures and the memorymanagement processes that form what we call the infrastructure forreal-time presentation of sensory information.

[0118] Risks and Risk Management. The risks inherent in this task arisefrom unrecognized or unresolved bottlenecks remaining in ourinfrastructure for real-time presentation of sensory information. Thisrisk is managed in our approach by thorough analysis of resourceallocation requirements prior to commitment in software of anyparticular data management configuration. Furthermore, subsequent tasksbuild on this infrastructure and therefore continue the process ofchallenging and reinforcing our approach to an infrastructure forreal-time presentation of sensory information.

[0119] Measures of Success. Completion of this task can be recognized bythe existence of a fully implemented and tested data management system.The level of success achieved for this task can be directly measured interms of data throughput relative to system requirements.

Task 2. Visual Representation of Smoke and Fire

[0120] Task Summary. The way in which sensory stimuli are presented inan ARBT scenario may or may not effect task performance by a student. Itis essential to capture the aspects of the sensory representations offire and smoke that affect student behavior in a training scenariowithout the computational encumbrance of those aspects that do notaffect behavior. For the purposes of providing sensory stimuli forfirefighter training, we need to know not only the spatial distributionand time evolution of temperature and hot gases in a compartment fire,but also the visible appearance of smoke and flame, along with soundsassociated with a burning compartment, taken over time. There are threetiers of attributes of fire and smoke:

[0121] First tier: location and extent

[0122] Second tier: opacity, luminosity, and dynamics

[0123] Third tier: illumination of other objects in the scene

[0124] Approach. Part of the rationale behind the problem identifiedabove is the degree to which the time- and 3D-space-dependent elementsof a desired scenario for a compartment fire can be precomputed. Thevisual representations of fire and smoke can be precomputed. In order todo so, and still retain real-time effects, the appearance of the fireand smoke from reasonable vantage points within the compartment would bedetermined. As a firefighter moves through a training simulation, theappropriate data need only be retrieved in real time to provide thenecessary visual stimulation.

[0125] The emission of visual light from flame and the scattering andabsorption of light by smoke is to be modeled. A zone-type fire modelcan be used to determine the location and extent of the smoke and flame.In addition to these quantities, the zone-type fire model also willyield aerosol densities in a given layer. Values for opticaltransmission through smoke can be calculated using a standard model suchas found in the CFAST (Consolidated Fire and Smoke Transport) model, orin the EOSAEL (Electro-Optical Systems Atmospheric Effects Library)code.

[0126] It is thought that the intermittent flame region in a fireoscillates with regularity, and that the oscillations arise frominstabilities at the boundary between the fire plume and the surroundingair. The instabilities generate vortex structures in the flame which inturn rise through the flame resulting in observed oscillations. For thepurposes of this description, the visual dynamics of flame can bemodeled from empirical data such as is known in the art.

[0127] Measures of Success. This task can be judged on the aesthetics ofthe visual appearance of the simulated fire and smoke. Ultimately, thevisual appearance of fire and smoke should be evaluated relative to theefficacy of an ARBT system.

Task 3. Position Anchoring

[0128] Task Summary. Augmented reality techniques rely on superimposinginformation onto a physical scene. Superposition means that informationis tied to objects or events in the scene. As such, it is necessary thento compensate for movement by an observer in order to maintain thegeometric relations between superimposed information and underlyingphysical structures in the scene.

[0129] Approach. Position sensors in the form of a head tracker can, inreal-time, calculate changes in location caused by movement of afirefighter within a training scenario. Virtual objects will be adjustedaccordingly to remain “fixed” to the physical world.

[0130] Risks and Risk Management. Rapid movements by an observer cancause superimposed information to lag behind the apparent motion ofobjects in the field of view. This lag may result in the feeling thatthe superimposed information is floating independent of the scene ratherthan remaining anchored to a specific position. In severe cases the lagin motion compensation may result in a form of simulator sickness whicharises when conflicting motion information is received by the brain. Inorder to minimize this effect, we can again consider the complexity ofthe visual presentation of augmented information. (It may also bepossible to essentially blank out the augmented information untilobserver movement stabilizes.)

[0131] Measures of Success. Anchoring virtual flame and smoke to aspecified position in a real room with minimal motion lag signals thecompletion of this task.

Task 4. Authoring Tools

[0132] Task Summary. The implementation of any sort of authoring toolfor instructors to create training scenarios is beyond the scope of thisdescription. However, because we do envision the creation of a prototypeauthoring system, this task is devoted to the investigation of issuesand characteristics involved. An authoring system typically takes theform of a visual programming interface over a modular toolkit offundamental processes. A training instructor can use an authoring toolto visually select and sequence modules to create the desired trainingcourse without ever having to resort to direct programming in somecomputer language such as C or FORTRAN.

[0133] Approach. Authoring tools do exist for construction of general,business-oriented, computer-based training. Examination of successfulattempts can serve as an instructive guide to specification of anauthoring system supporting ARBT for firefighters.

[0134] Risks and Risk Management. Although there is no risk, per se,inherent in this task, authoring any real-time system is problematic. Anauthoring system relies on the existence of independent modules that areexecuted through a central control facility. If the control modulehandles all data traffic, then bottlenecks may occur that would notnecessarily exist in an optimized, real-time system.

[0135] Measures of Success. This task leads into development forinstructors of an authoring system for an ARBT system for firefighters.The measure of success then lies in the coverage of issues pertaining to

[0136] authoring of real-time systems

[0137] commercially available authoring tools or systems

Task 5. ARBT Technology Demonstration

[0138] Task Summary. The previous tasks herein developed the pieces ofan augmented reality fire simulation. It remains to pull everythingtogether into a coherent demonstration to show the suitability of theselected technologies to the delivery of training to firefighters.

[0139] Approach. A scenario consisting of a real room and virtual fireis to be constructed, and a problem solving situation will be presentedto prospective trainees.

[0140] Risks and Risk Management. The obvious risk is that the virtualfire and smoke training demonstration scenarios do not achieve adequaterealism to an experienced firefighter.

[0141] Measures of Success. The real measure of success for this tasklies in the realism perceived by a trainee. In order to judge thesuccess of the demonstration, the users will evaluate the effectivenessof the simulation.

[0142] Appendix A

[0143] The descriptions in this Appendix contain parameters that may beused to describe the behavior of particle systems used to represent thefollowing phenomena:

[0144] Flames

[0145] Smoke plume

[0146] Smoke with random motion to be used in the upper layer

[0147] Steam

[0148] Water spray from a vari-nozzle

Default Fire Parameters

[0149] System Type: Faded, Directional Quads

[0150] Emitter Shape: Rectangular

[0151] Global Force Vector (lbf): 0.0, 1.00, 0.0

[0152] Particle Mass: 0.2 lbm

[0153] Mass Variance: 0.0

[0154] Yaw: 0.0 radian

[0155] Yaw Variance: 6.28 radian

[0156] Pitch: 0.0 radian

[0157] Pitch Variance: 0.05 radian

[0158] Initial Speed: 0.65 ft/s

[0159] Initial Speed Variance: 0.05 ft/s

[0160] Emission Rate: 750 particles/sec

[0161] Emission Rate Variance: 500 particles/sec

[0162] Life Span: 1.3 sec

[0163] Life Span Variance: 0.1 sec

[0164] Start Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0165] Middle Color (RGBA): 1.0, 1.0, 1.0, 1.0

[0166] End Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0167] Random Force: 0.45 lbf

[0168] Start Scale: 1.0

[0169] End Scale: 1.0

Default Smoke Parameters

[0170] System Type: Billboard

[0171] Emitter Shape: Rectangular

[0172] Global Force Vector (lbf): 0.0, 1.00, 0.0

[0173] Particle Mass: 0.2 lbm

[0174] Mass Variance: 0.05

[0175] Yaw: 0.0 radian

[0176] Yaw Variance: 6.28 radian

[0177] Pitch: 1.2 radian

[0178] Pitch Variance: 0.393 radian

[0179] Initial Speed: 0.2 ft/s

[0180] Initial Speed Variance: 0.0 ft/s

[0181] Emission Rate: 10.0 particles/sec

[0182] Emission Rate Variance: 0.25 particles/sec

[0183] Life Span: 3.25 sec

[0184] Life Span Variance: 0.25 sec

[0185] Start Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0186] Middle Color (RGBA): 1.0, 1.0, 1.0, 0.85

[0187] End Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0188] Random Force: 0.4 lbf

[0189] Start Scale: 0.105

[0190] End Scale: 4.2

Default Layer Smoke Parameters

[0191] System Type: Billboard

[0192] Emitter Shape: Rectangular

[0193] Global Force Vector (lbf): 0.0, 0.025, 0.0

[0194] Particle Mass: 0.2 lbm

[0195] Mass Variance: 0.05

[0196] Yaw: 0.0 radian

[0197] Yaw Variance: 6.28 radian

[0198] Pitch: 1.2 radian

[0199] Pitch Variance: 0.393 radian

[0200] Initial Speed: 0.0 ft/s

[0201] Initial Speed Variance: 0.0 ft/s

[0202] Emission Rate: 35.0 particles/sec

[0203] Emission Rate Variance: 5.0 particles/sec

[0204] Life Span: 4.0 sec

[0205] Life Span Variance: 0.5 sec

[0206] Start Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0207] Middle Color (RGBA): 1.0, 1.0, 1.0, 1.0

[0208] End Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0209] Random Force: 0.35 lbf

[0210] Start Scale: 1.75

[0211] End Scale: 3.75

Default Steam Parameters

[0212] System Type: Billboard

[0213] Emitter Shape: Rectangular

[0214] Global Force Vector (lbf): 0.0, 0.4, 0.0

[0215] Particle Mass: 0.2 lbm

[0216] Mass Variance: 0.05

[0217] Yaw: 0.0 radian

[0218] Yaw Variance: 6.28 radian

[0219] Pitch: 1.2 radian

[0220] Pitch Variance: 0.393 radian

[0221] Initial Speed: 0.6 ft/s

[0222] Initial Speed Variance: 0.0 ft/s

[0223] Emission Rate: 50.0 particles/sec

[0224] Emission Rate Variance: 10.0 particles/sec

[0225] Life Span: 2.5 sec

[0226] Life Span Variance: 0.5 sec

[0227] Start Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0228] Middle Color (RGBA): 1.0, 1.0, 1.0, 0.45

[0229] End Color (RGBA): 1.0, 1.0, 1.0, 0.0

[0230] Random Force: 0.25 lbf

[0231] Start Scale: 0.7

[0232] End Scale: 2.8

Default Water Stream Parameters

[0233] System Type: Surface

[0234] Emitter Shape: Spherical

[0235] Global Force Vector (lbf): 0.0,-2.11, 0.0

[0236] Particle Mass: 0.0656 lbm

[0237] Mass Variance: 0.005

[0238] Initial Speed: 162.0 ft/s

[0239] Initial Speed Variance: 0.0 ft/s

[0240] Life Span: 4.5 sec

[0241] Life Span Variance: 0.4 sec

[0242] Start Color (RGBA): 0.65, 0.65, 1.0, 1.0

[0243] Middle Color (RGBA): 0.65, 0.65, 1.0, 0.5

[0244] End Color (RGBA): 0.75, 0.75, 1.0, 0.0

[0245] Random Force: 0.15 lbf

[0246] Although specific features of the invention are shown in somedrawings and not others, this is for convenience only, as each featuremay be combined with any or all of the other features in accordance withthe invention.

[0247] Other embodiments will occur to those skilled in the art and arewithin the following claims.

What is claimed is:
 1. A method of simulating flow of an extinguishingagent from a fire hose or other agent application means for graphicaldisplay, comprising: employing at least one particle system to simulatephysical behavior of an extinguishing agent stream; organizing particlesinto regularly spaced rings; using particle locations as polygonvertices to create a surface; texture mapping said surface; and movingsaid texture map in the direction of extinguishing agent flow tosimulate extinguishing agent flow.
 2. The method of claim 1 in whichthere are surfaces in the display, and wherein the intersection of awater stream with a surface generates steam particles.
 3. The method ofclaim 1 in which there is a flame base grid including sub-areas, and inwhich flames are displayed emanating from the flame base, and in whichthe intersection of the extinguishing agent stream with the flame basegrid sub-area is determined, and used to reduce the height of the flamesemanating from that sub-area.
 4. The method of claim 1 in which thereare polygonal models in the display, and wherein the extinguishing agentparticles' collision with a polygonal model are determined, and thedirection of movement of the extinguishing agent particles is changedaccordingly based on particle elasticity.
 5. The method of claim 1 inwhich two rings of particles are emitted to create an outer and innersurface.
 6. The method of claim 5 in which the outer surface is given adifferent, more transparent texture than the inner surface in order tomask the surface edges.
 7. The method of claim 4 in which collision isdetermined by testing every particle with all polygons that they maycollide with.
 8. The method of claim 4 in which selected particles aretested for collision with selected polygons in order to improve speed.9. The method of claim 8 in which space is partitioned into a 3-D gridin order to limit the number of polygons and particle pairings that aretested for collision.
 10. The method of claim 8 in which space ispartitioned using a binary space partition (BSP) tree approach in orderto determine which particles to test for collision with which polygons.11. The method of claim 8 in which space is partitioned using an octreespace partition (OSP) tree approach in order to determine whichparticles to test for collision with which polygons.
 12. The method ofclaim 4 in which collision is determined by setting axis-aligned minimumand maximum bounds for particle motion.
 13. The method of claim 1further comprising employing another particle system to represent waterdroplets.